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DEVELOPMENT OF EVALUATION CRITERIA 
AND A PROCEDURE FOR ASSESSING 
PREDICTIVE CAPABILITY AND CODE PERFORMANCE 

S. J. Lin, S. L. Barson, M. M. Sindir, and G.H. Prueger 


ABSTRACT 

Computational Fluid Dynamics (CFD), because of its unique ability to predict complex 
three-dimensional flows is being applied with increasing frequency in the aerospace 
industry. Currently, no consistent code validation procedure is applied within the 
industry. Such a procedure is needed to increase confidence in CFD and reduce risk 
in the use of these codes as a design and analysis tool. This final contract report 
defines classifications for three levels of code validation, directly relating the use of 
CFD codes to the engineering design cycle. Evaluation criteria by which codes are 
measured and classified are recommended and discussed. Criteria for selecting 
experimental data against which CFD results can be compared are outlined. A four 
phase CFD code validation procedure is described in detail. Finally, the code 
validation procedure is demonstrated through application of the REACT CFD code to a 
series of cases culminating in a code to data comparison on the Space Shuttle Main 
Engine High Pressure Fuel Turbopump Impeller. 

1.0 INTRODUCTION 

Applications such as the National Launch System (NLS), the National Aero-Space 
Plane (NASP), or any of the single stage to orbit (SSTO) concepts being considered 
require advanced computational modeling to define vehicle and propulsion system 
performance over the nominal flight envelope and to test sensitivities to off-nominal 
conditions. Computational Fluid Dynamics (CFD) is unique in its ability to predict 
complex three-dimensional flows associated with vehicles and their propulsion 
systems. Judicious application of CFD in the design cycle can minimize test 
requirements, aid in designing better tests, and help to better interpret test data. 
Additionally, CFD can be used effectively in extrapolating to new operating conditions 
for which no test capability exists. Thus, CFD is playing an increasingly important role 
in the design of new space vehicles and their propulsion systems. CFD codes have to 
be systematically validated to increase confidence and reduce risk in their use for 
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design and analysis. The subject of CFD code validation is gaining recognition as a 
topic of importance and has been the subject of several recent publications^ ®). 

CFD at Rocketdyne is a key analysis tool, regularly used in the engineering design 
process. Five major CFD codes have been applied to a variety of problems on major 
programs such as the Space Shuttle Main Engine (SSME), the National Launch 
System (NLS), and the National Aero-Space Plane (NASP). CFD results are regularly 
used as the basis for design decisions. Thus, code validation is of considerable 
interest. This broad range of experience has provided insight into the practical issues 
surrounding CFD code validation. Some observations and key lessons learned are 
summarized below: 

1 ) A general code validation procedure for all codes and applications can be 
developed. 

2) Specific, quantitative evaluation criteria are highly application dependent and it 
is not possible to define a single general set of validation criteria. 

3) Quantitative validation is only meaningful within limited classes of applications. 

4) The level of validation appropriate depends on the intended use of the CFD 
predictions. 

5) The validation process must be realistically achievable within the engineering 
environment. In this environment, pressure to apply a code and produce 
results before validation is complete may be significant. Thus, the validation 
process must be flexible, allow for varying levels of validation, and incremental 
improvement as time and funding permit. 

The objective of this effort is to define a comprehensive procedure with associated 
criteria through which all aspects of CFD codes can be validated in a consistent 
manner. These aspects include basic programming, solution methodology, code 
numerics, and physical models, as applied through the integrated CFD code. The 
goals of this approach are to improve and quantify understanding of the CFD code 
predictive capability, to establish consistent application techniques within classes of 
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similar problems, and to increase confidence in the use of CFD tools for engineering 
problems. 

This contract report includes general discussion on the topic of CFD code validation. 
Code classifications are defined, directly relating use of CFD codes to engineering 
design. Evaluation criteria are developed and, because code validation depends on 
comparisons of CFD predictions with experimental data, criteria for experiments are 
also outlined. A four phase CFD code validation procedure is recommended and 
described in detail. Finally, the code validation procedure is demonstrated by 
applying it to REACT (Rocketdyne Elliptic Analysis Code for Turbomachinery). 

1.1 Code Validation Cl assifications 

A primary goal of this effort is to encourage consistent application of CFD codes in 
engineering design. This will result in increased confidence in the use of these tools 
and reduce associated risks. However, CFD methods can be effectively applied with 
widely differing levels of accuracy. Early in the design cycle, during the conceptual 
definition phase, demands placed on the code may be limited to proper prediction of 
qualitative trends. Late in the design cycle, during the detailed design phase, 
extensive demands may be made of the codes, requiring detailed and accurate 
flowfield prediction. 

Validation may be time consuming. In the engineering environment, pressure may be 
strong to apply a code before it is thoroughly validated. It is appropriate that a range of 
code validation be allowed to accommodate engineering needs. CFD codes validated 
according to defined procedures may be classified based on demonstrated 
capabilities. Once classified, codes should be applied only within these limits. 

MehtaO) defined five classifications for validated codes. To meet engineering needs, 
a simplified approach is proposed defining three levels of code validation. 


1) Onnrflntual Design-Validated Code . Before a code can be considered 
validated for use in conceptual design, the following conditions must be met: 
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a. Basic code methodology must be reviewed and considered relative to the 
end application. 

b. A study of code operability, exercising all options relevant to the end 
application must be conducted. 

c. A systematic determination of numerical accuracy must be completed along 
with successive grid refinement studies. 

d. Physical models to be employed in the final application must be 
quantitatively verified through comparison with data from benchmark 
experiments. 

e. The entire code must be exercised to demonstrate on simple, but relevant 
problems the ability to produce proper qualitative results. 

With completion of these activities the code may be considered to be 
conceptual design-validated. The range of applicability is restricted to a 
class of problems similar to that for which the validation was conducted. 
Extension to significantly different problems (e.g., involving new physics) 
requires further validation for parts of the code not previously verified. 

2) Preliminary Design-Validated Code . For a code to be considered validated for 
use in preliminary design activities all of the conceptual design validation 
requirements outlined above must be met. Additionally, the following conditions 
must also be met: 

a. Computed results for problems similar to that of interest must quantitatively 
agree with experimental data. Global performance quantities computed 
from CFD results must show a level of agreement consistent with 
established evaluation criteria. These criteria depend on the end 
application and must be established by those using the computational 
results (i.e., analyst and designer). 
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b. The accuracy and limitations of experimental data used for comparisons 
must be known and well understood. 

c. Effects of grid distribution on prediction of global performance quantities 
must be established. 

With completion of these activities the code may be considered to be 
preliminary design-validated. The range of applicability is restricted to a 
class of problems similar to that for which the validation was conducted. 
Extension to significantly different problems requires further validation. 

3) Detail Design-Validated Code . A code is considered to be validated for use in 
detailed and final design activities if, in addition to satisfying all qualifications 
set forth in 1 and 2 above, the following conditions are met: 

a. Comparisons of computed results with available hardware test data show 
that the code is able to adequately model all physical effects relevant to the 
problem of interest. 

b. Effects of grid density on the prediction of detailed flowfield and surface 
quantities must be established. 

With completion of these activities the code may be considered to be detail design- 
validated. The range of applicability is restricted to a class of problems similar to that 
for which the validation was conducted and extension to -significantly different 
problems requires further validation. 

1.2 Code Evaluation Criteria 

Evaluation criteria are the metrics against which a code is judged. These criteria can 
be customized according to the above defined code validation classifications to 
provide the degree of confidence required for each phase in the engineering design 
process (conceptual, preliminary, detail design). Once a code has been classified at a 
given level, it can then be used confidently within that phase of the design cycle. 
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Ideally, end users would construct a complete set of criteria to meet their needs over all 
design phases. For example, key criteria would be identified and quantified for each 
validation phase. Table 1 represents the necessary information generically. 


1.3 Error Assessment 

Errors associated with CFD codes can arise from many sources. These include code logic, 
numerical methods, and physical models (e.g., turbulence, chemistry). These errors need to 
be systematically identified, understood and, where possible, reduced before CFD can be 
used confidently. 

Various measures can be taken to check the code logic. Independently programmed, but 
logically identical modules can be substituted and cross-checked. Grid studies can be 
conducted to ensure that successive refinement produces a correct grid independent 
solution. Consistency checks can be performed for established physical properties such as 
symmetry (e.g., does an airfoil of symmetric section at zero angle of attack produce lift). 
These checks are incorporated into early phases of the proposed validation procedure. 

Table 1. Generic Representation of Criteria for Three Validation Phases 


Criteria 

Validation Phase 

Conceptual 

Preliminary 

Detail 

A 

Qualitative 

10% 

5% 

B 

Qualitative 

20% 

10% 

C 

10% 

5% 

2% 


Errors associated with numerical methods are inherent in every computational methodology. 
Discretization errors are associated with having a finite number of grid points, truncation, and 
coordinate transformation. Errors are also associated with the solution algorithm, generally 
an iterative procedure, and incomplete convergence. For this type of error, comparisons with 
an exact analytical solution may provide more insight than comparison with experiments. 
Comparison with high quality experimental data is, of course, the ultimate test of a code, but 
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Errors associated with numerical methods are inherent in every computational 
methodology. Discretization errors are associated with having a finite number of grid 
points, truncation, and coordinate transformation. Errors are also associated with the 
solution algorithm, generally an iterative procedure, and incomplete convergence. For 
this type of error, comparisons with an exact analytical solution may provide more 
insight than comparison with experiments. Comparison with high quality experimental 
data is, of course, the ultimate test of a code, but should only be done after numerical 
errors have been identified, understood and, where possible, reduced. 

Discretization error represents the difference between a well converged solution of the 
discretized equations and the exact solution. Discretization error (for complex 
flowfields) can be quantified by either obtaining an "exact" solution through successive 
grid refinement (usually an expensive and time consuming task) or by using 
Richardson's method that expresses the error as a Taylor series in a parameter, h, 
representative of the grid size. It can be shown that for first order accuracy the error, 
eh, between two grids h and 2h (a grid twice as coarse), can be estimated as, 

eh - 4>h - <t>2h 


where 0 represents the converged solution for a given grid. For a second order 
solution the error becomes, 

<J>h - <t>2h 
eh 3 

Prudent use of grid sensitivity studies combined with the Richardson method can be 
successfully used to estimate errors due to discretization. 

Most methods used in CFD utilize iterative procedures. Typically, iteration is stopped 
when the difference between two successive iterates, measured by some norm, is less 
than a preselected level. Unfortunately, the convergence error, defined as the 
difference between the current iterate and the exact solution of the discretized 
equations depends not only on the difference between successive iterates, but also on 
the rate of convergence. It is possible to derive an expression for the error and use 
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this as the basis of a convergence criterion. It can be shown^ that the error e is a 
function of the principle eigenvalue, X -\ , as well as the difference between two 
successive iterates, <}> n+ 1 and <j> n ; this is given as, 



where 4> represents the solution. Various parameters can be used as <{> to monitor 
convergence. These parameters may either be taken directly from the solution or 
calculated. The choice is application dependent. 


Finally, errors associated with physical modeling (turbulence, chemistry) must be 
addressed. These are typically the most difficult to identify and reduce. While it is 
generally accepted that the Navier-Stokes equations are applicable to continuous fluid 
physics, the ability of CFD to predict complex flow physics may strongly depend on 
modeling of turbulence and chemistry effects. Therefore, the range of application for a 
particular CFD code is limited by the physical models employed. Careful 
quantification of physical model limitations must be carried out. Comparison with 
benchmark experiment data should be performed relatively early in the validation 
process. 



For all but the most fundamental flow cases experimental data provide the only means 
for evaluating whether CFD solutions are correct or not. Further, these data provide 
the only means for assessing an absolute level of agreement between CFD and the 
true flow physics. Because of this dependence on experimental data it is essential that 
experiments selected for CFD code validation be of the highest quality possible. 
Settles and Dodson( 9 ) divided criteria for selecting validation experiments into two 
categories: "necessary" and "desirable". Experiments are first measured against the 
"necessary" set of criteria. Those experiments that do not satisfy all of these criteria 
should not be used for code validation. Experiments satisfying the first set of 
requirements should then be judged against the second set of "desirable" criteria. The 
best experiments will meet all of the "necessary" criteria and should satisfy many of the 
"desirable" criteria. Adapting and generalizing the recommendations of Settles and 
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Dodson, "necessary" and "desirable" criteria are listed below, roughly in order of 
importance. 

"Npnessarv" Criteria 

1 ) Applicability to Problem of Interest . Candidate experiments must be relevant to 
the end application. For more fundamental flows, experiments should ideally 
represent a single flow feature typical of the final application. For more 
complex flows, experiments should represent two or more flow features typical 
of the final application. 

2) Wall-Defined Experimental Boundary Conditions . Candidate experiments 
must provide sufficient and accurate information at all flow boundaries to allow 
accurate CFD modeling. Typical data should include detailed definition of 
inflow and outflow conditions including velocity distributions, pressure, 
temperature, total conditions (as applicable), and wall temperatures. 

3) WPll-Defined Experimental Error Bounds . The experimenter must provide a 
substantiated analysis of the accuracy and repeatability of the data. This error 
analysis should be represented through error bars on the data. Without this 
information comparisons between CFD and test data can not be accurately 
interpreted. 

4) Splf-Cnnsigtpnrv of Data . Results from a given experiment must not be 
contradictory. If such results are found, they must be either resolved, preferably 
through direct contact with the experimenter, or the results should not be used 
for CFD code validation. 

5 ) Adpgnate Documentation of Data . Experimental data must be documented 
with sufficient detail and clarity to allow for direct numerical comparisons. Data 
should be available in a tabular form that can be easily compared and cross- 
plotted with computational results. Data available only in the form of plots must 
be sufficiently legible that numerical values can be ascertained well within 
stated error bounds. 
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6) Attenuate Spatial Resolution of Data . Data must be provided in sufficient detail 
to adequately resolve key flow features. This is particularly important for more 
fundamental flows where basic physical models within the CFD code are to be 
evaluated. It is recognized that for more complex flows, data are typically less 
available. 

"Desirable" Criteria 

1) Data for Phvsinal Models . Experiments conducted with the intention of 
providing data on basic physical phenomena (typically represented by physical 
models within CFD codes) should include more than simple mean-flow 
measurements. Appropriate data might include Reynolds stresses or spatial 
distribution of chemical species. 

2) Nonintrusive Instrumentation . Nonintrusive measurements are the preferred 
data acquisition technique. Characteristic of this type of instrumentation, 
questions of relative error are largely alleviated. 

3) Redundant Measurements . Redundant measurements provide a means for 
easily verifying the "necessary" self-consistency criteria. Ideally, data should 
be taken to provide alternate methods of measuring key flow features and to 
verify basic modeling assumptions (e.g., replication of data mirrored across 
symmetry plane substantiates use of the CFD symmetry modeling assumption) 

4 ) Flow Structure and Physics . Measurements that reveal flow structure are 
strongly desired. Relatively new techniques such as planar laser-induced 
fluorescence (PLIF) provide nonintrusive measurements in two-dimensional 
cuts through the flowfield. This allows for direct high level comparison of CFD 
predictions with spatially accurate flow measures. Visualization techniques 
typically used to postprocess CFD results can similarly be applied to measured 
data improving qualitative understanding of the flow as well. 
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2.0 CFD CODE VALIDATION PROCEDURE 


The general code validation procedure developed is described in this Section. This 
procedure may be used with any CFD code and can be customized for any application 
of interest. Because quantitative evaluation criteria are application dependent they 
have been uncoupled from the general procedure. The proposed validation 
procedure can be realistically performed within typical constraints of the engineering 
environment. This process is flexible, allowing for varying levels of validation to be 
performed and incrementally upgraded as time and funding permit. 

Because the procedure must ultimately be customized for a given class of 
applications, requirements directly related to the end application must be identified 
first. One must assess the level of validation required (i.e, for conceptual, preliminary, 
or detail design). Appropriate criteria based on engineering design methods must be 
established for the selected level. These criteria will typically be expressed in terms of 
the level of agreement required for conceptual (qualitative trends), preliminary (global 
performance values), or detail design (specific flow features and values) code 
validation. 

Having established code evaluation criteria, one must select appropriate fundamental 
flow cases, benchmark experiments, and quality tests against which CFD predictions 
can be compared. Selected cases must directly represent one or more features 
characteristic of the end application. To ensure this, one must study that problem and 
consider all relevant features. The end application is then successively decomposed 
into a series of less complex problems for which quality data exist. 

This successive decomposition occurs over four steps as an integral part of the 
validation procedure. The four phase validation procedure is illustrated in Fig. 1. 
Phases 1 through 4 represent increasing levels in flow and geometric complexity. 
Phase 1 includes fundamental flows only. Phase 4 includes complex flows that 
directly represent the end application. Availability of data generally decreases as the 
flow complexity increases. Often, the quality of that data decreases as well. As one 
progresses through each validation phase, additional information about the CFD code, 
as applied to the end application, is obtained. Information learned in Phase 1 is 
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PHASE 1 
UNIT PROBLEM 

PHASE 2 

BENCHMARK CASES 

PHASE 3 

SIMPLIFIED PARTIAL 
FLOWPATH 

PHASE 4 

ACTUAL HARDWARE 

! • SINGLE FLOW 
FEATURE 

1 • ANALYTIC SOLUTION 
OR HIGH FIDELITY 
COMPUTATIONAL 
SOLUTION (DNS) 
AVAILABLE 

• MORE THAN ONE 
FLOW FEATURE 

• SIMPLE FLOW 
PHYSICS 

• BENCHMARK 
EXPERIMENT 
DATA 

• MULTIPLE RELEVANT 
FLOW FEATURES 

• ACTUAL FLOW 
PHYSICS 

• HIGH QUALITY 
TEST DATA 

• COMPLETE FLOW 
PHYSICS 

• HARDWARE TEST 
DATA 


INCREASING GEOMETRIC AND FLOW COMPLEXITY 


. rp m 

- x: -• 


• RUN UNIT PROBLEMS 

• VERIFY INTEGRITY 

• ASSESS ACCURACY, 
CONVERGENCE, 

AND FUNCTIONALITY 

• RUN BENCHMARK 
CASES 

• ASSESS PHYSICAL 
MODELS 

• ESTABLISH GRID 
DISTRIBUTION 
REQUIREMENTS 

• RUN SIMPLIFIED 
PARTIAL FLOWPATH 

• ASSESS AGREEMENT 
WITH DATA 

• ESTABLISH GRID 
DISTRIBUTION 
REQUIREMENTS 

• RUN AC 
CONFIC 
■ COMPA 
TEST D 

TUAL 
3U RATION 
RE WITH 
ATA 

DECREASING DATA AVAILABILITY AND ACCURACY 



Figure 1. Four Phase Code Validation Procedure 


applied in Phase 2 an so on. Ultimately, an extensive knowledge base is developed 
in support of the final application. 

2.1 Phase i - Unit Problems 

Relevant unit problems, based on successive decompositions of the end application, 
are identified in Phase 1 . Unit problems are characterized by a single dominant flow 
feature and have available analytical solutions. In Phase 1, the CFD code is 
exercised on several unit problems, each representing one basic flow feature of the 
end application. This phase acts as a final code verification in which fundamental 
code characteristics are thoroughly understood and documented. Basic code 
methodology is considered in terms of its applicability to the problem of interest. All 
aspects of the code relevant to the end application are exercised to verify accuracy, 
functionality, and convergence characteristics. At least one unit problem is selected to 
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extensively test basic code logic through tests previously suggested (substitution of 
key code modules, tests for symmetry, etc.). Additionally, systematic grid sensitivity 
studies are conducted, both to assess relative error and to provide guidance in 
specifying computational grids for more complex flow cases. 

2.2 Phase 2 - Benchmark Cases 

Relevant benchmark cases, based on successive decompositions of the end 
application, are identified in Phase 2. These benchmark cases are relatively simple as 
compared with the final application, but are characterized by more than one flow 
feature. Phase 2 cases should include basic physics relevant to the final application. 
Physical models within the CFD code are exercised to verify operability and to quantify 
accuracy relative to the benchmark data. Only data from the highest quality 
experiments should be used for comparisons with CFD solutions. Grid sensitivity 
studies are conducted to assess the level of refinement necessary to capture key 
physical effects. Error assessment techniques previously discussed are used as a 
guide. Lessons learned from Phase 1 should be applied to Phase 2. Overall, fewer 
cases will be run in Phase 2 than were run in Phase 1. A code validated though 
Phase 2, satisfying all established criteria may be considered validated for conceptual 
design studies. 

2.3 Phase 3 - Simplifie d Partial Flowpath 

Test cases selected for Phase 3 are moderately complex. These cases are 
simplifications of the final validation case, each representing multiple geometric or flow 
features of the final application. Actual flow physics of the final application should be 
reasonably well represented by these cases. At this level of complexity, high quality 
data may be difficult to obtain. Data should be selected according to the criteria 
previously described, but these criteria may be relaxed slightly if needed. A different 
type of grid sensitivity study is performed during the Phase 3 validation. An 
assessment on the effect of variations in grid topology and grid clustering is done to 
provide guidance for the end application. Again, the goal is to establish grid 
requirements necessary to capture key physical effects. Knowledge gained in 
Phase 2 sensitivity studies should prove to be useful. Relatively few cases will be run 
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in Phase 3. A code validated though Phase 3, satisfying all established criteria may 
be considered validated for preliminary design studies. 

2.4 Phase 4 - Actual Hardware 

Cases for Phase 4 should be selected from tests conducted using actual hardware. 
Thus, all of the relevant geometric and physical effects should occur simultaneously. 
Test data may be less available and of lower quality than that of earlier phases. 
Selection criteria should be carefully reviewed to allow choice of the best data sets 
and to identify where deficiencies in the data may exist. The knowledge base 
developed in Phases 1 through 3 should be applied in Phase 4. The most appropriate 
physical models, best grid topology, and an appropriately refined grid should be used. 
It is likely that only one or two cases will be run in Phase 4 of the validation procedure. 
A code validated though Phase 4, satisfying all established criteria may be considered 
validated for detail design studies. 

2.5 Incrementally Extending Code Validation 

As a CFD code is validated to different levels for a given application or extended to 
new applications, a database will be developed and gradually extended. This 
database will include selected analytical cases, benchmark experiment data, high 
quality test data, hardware test data, and associated CFD solutions. Therefore, 
extending an existing validation effort to either the next level or for a new application is 
relatively easy. As depicted in Fig. 2, much of the work may already be complete and 
comparatively few cases may need to be run. 

3.0 DEMONSTRATION OF THE CODE VALIDATION PROCEDURE 

3.1 Code and Application Selected 

The proposed code validation procedure is fairly detailed and it is most easily 
illustrated by example. The following sample validation exercise was performed 
primarily for illustrative purposes. 
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Figure 2. Building Block Approach to Develop Validation Database 

3.1.1 Identify Code to be Validated 

The evaluation procedure discussed can be used to evaluate any CFD code. 
REACTOR 12) (Rocketdyne Elliptic Analysis Code for Turbomachinery) was 
selected for this demonstration effort. 

The REACT code is a general purpose 2-D/3-D full Navier-Stokes code. REACT 
operates in generalized coordinates and uses a second-order correct finite volume 
discretization scheme. Various solvers including conjugate gradient, Stone's strongly 
implicit procedure, and ADI techniques are available. The code offers various 
turbulence models such as the standard k-e, low Reynolds number k-e, and multiscale 
k-e. The REACT methodology is applicable for flow conditions ranging from 
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incompressible to low supersonic flow. The code accommodates a variety of 
boundary conditions including multiple inlets and outlets, planes of symmetry, spatial 
periodicity, and internal obstacles. Geometric complexities may be accommodated 
through a multiple zone approach. 

REACT has been used to solve many flow problems. Solutions have been obtained 
for virtually every component and type of flow encountered in a turbopump including 
inducers, impellers, crossovers, volutes, turbine cascades, cavity flows, and bearing 
flows. 

3.1.2 Select Final Application and Decompose Over Four Phases 

Rocketdyne holds a strong interest in the application of CFD to the design and 
analysis of turbopumps. In association with the Rocketdyne role as developer of the 
Space Shuttle Main Engine (SSME), a series of nonintrusive measurements were 
taken on a SSME high pressure fuel turbopump (HPFTP) impeller. The availability of 
quality data for a complex piece of flight hardware, combined with interest in 
turbomachinery makes this an ideal validation case. Thus, the end application was 
identified and the goal was set to validate REACT for impeller applications. 

Figure 3 illustrates the approach of successive decomposition. Given that the Phase 4 
test case is an impeller, key flow features were identified. Impellers are characterized 
by highly three-dimensional geometry, strong curvature, and high rotational speeds. 
For this impeller, there are three partial blades between the full blades (Fig. 4.). 

Phase 3 cases selected represent the impeller as two types of simplified flowpaths, 
each less complex than the complete impeller problem, but still with multiple flow 
features represented in the impeller. Flow within blade passages of a shrouded 
impeller were conceptually simplified and represented as flow through a rotating 
curved duct. Flow over the partial blades was reduced to flow over a three- 
dimensional turbine blade cascade. 

Phase 2 cases were selected by decomposing those from Phase 3. The rotating 
curved duct was decomposed into flow in curved non-rotating ducts and flow about a 
rotating disk. Flow over a 3-D turbine blade cascade was represented by turbulent 
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PHASE 1 
UNIT 

PROBLEMS 

• flat plate 

• straight duct 

• diffuser 

• sudden 
contraction (lam.) 

• backward facing 
step (lam.) 

• driven cavity 

• rotating 
concentric 
cylinders (Taylor- 
Couette flow) 


PHASE 2 

BENCHMARK 

CASES 

• square duct with 
90° bend 

• S-shapedduct 

• backward facing 
step (turb.) 

• orifice flow (turb.) 

• flow around 
confined bluff 
bodies 

• 2-D turbine cascade 

• rotating disk 


PHASE 3 

SIMPLIFIED 

FLOWPATHS 


• 3-D turbine blade 
cascade 

• rotating curved 
duct 


PHASE 4 

ACTUAL 

HARDWARE 


. SSMEHPFTP 
impeller (2 sets 
partial blades) 


Figure 3. Successive Decomposition of impeller 
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flow over a variety of obstacles including backward facing steps, around confined bluff 
bodies, through extreme contraction and expansion of an orifice, and over 2-D turbine 
blades. 

Finally, Phase 1 cases were selected by simplifying the Phase 2 cases one last time. 
The curved duct and rotating disk cases were further simplified. Straight duct flow, 
flow over flat plates, and fundamental cases with rotation (e.g., Taylor-Couette flow) 
were examined. The flows over obstacles were simplified to first look at laminar cases, 
removing the uncertainty of turbulence models. 

Of the cases completed and represented in Fig. 3, the following were chosen to 
highlight various parts of the procedure: 

1 ) Straight passage flow (analytic solution) 

2) Square duct with 90° elbow (benchmark experiment data 1 3 ) 

3) SSME 3-D turbine cascade (test data 14 - 15 ), 

4) SSME HPFTP impeller (test data 16 ). 

Because Case 1 has an analytic solution, flow variables are known exactly. Test data 
for cases 2 through 4 include flow quantities at the boundaries. Additionally, case 2 
had streamwise and radial velocity distribution measurements at several locations. 
Test data for case 3 also included static pressure distribution on the blade surface. An 
estimate of the turbine efficiency bias and precision limits was performed and was 
estimated to be 0.7% of the efficiency. 

Test data for case 4 includes absolute and relative velocity and flow angle in several 
planes downstream of the impeller. The velocity measurements at the inlet plane and 
discharge of the impeller were completed with a L2F measurement system. This 
allowed for a highly accurate non-intrusive method of measuring the impeller inlet and 
discharge velocities. A plane approximately 1 inch upstream of the impeller was 
measured to provide a good inlet condition to the CFD model. Three planes 
downstream of the impeller were measured, these were at 5.570, 5.701, and 
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5.833 inches. The 5.570 inch plane was to measure the velocities just at the exit of 
the impeller discharge, the 5.833 inch plane was selected as this would be the 
standard location of the downstream component , and the 5.701 inch plane was to 
gain more data for validation. 

Cases 3 and 4 do not strictly satisfy all requirements set for benchmark experiment 
standards but are representative of some of the better data available. Considering the 
complexity of these flows, these data sets are more than sufficient for the present 
purpose of demonstrating the code evaluation procedure. 

3.1.3 Establish Code Evaluation Criteria 

The ultimate purpose of code validation is to establish a degree of confidence in the 
CFD code as applied in the design process. The level of predictive capability must be 
quantified in terms that are useful to the design engineer. For impeller design, a 
variety of analysis tools are employed during the course of the design cycle. 
Traditional (non-CFD) tools have been applied for many years over all design levels. 
A typical accuracy for these tools might be on the order of 10%. Consequently, test 
data are required for detail design and final quantification of performance. 

For the conceptual design phase, CFD results must demonstrate the correct qualitative 
trends. Error between test data and predicted results may, for particular parameters, 
be large (e.g., on the order of 30%). Because the goal in this design phase is to 
assess the merit of one design relative to another, larger errors are generally 
acceptable as long as predicted trendsTrom one design to the next are correct. 

For preliminary design of an impeller, global parameters should be predicted with 
relatively good accuracy. Two key parameters used to quantify impeller performance 
are efficiency and head rise. Impeller efficiency should be predicted within 1-2% and 
head rise should be within about 10%. Specific flow parameters such as velocity 
magnitude and flow angle should be predicted within about 5% and 1°, respectively. 
The flow split between passages should be within 5%. 

To provide detailed design data and minimize (or ultimately eliminate) the need for test 
data, accuracy should generally be on the order of the test data or better. Of the test 
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data obtained for the HPFTP impeller velocity magnitude and flow angle, error bands 
were quoted by the experimental group to be ±1% and ±0.5°, respectively. These 
values were, therefore set as criteria for CFD predictions at the detail design level. 
The flow split between passages should be within 2%. Agreement outside of these 
bands implies that, while CFD may be used for detailed design, some testing may still 
be required. Of the global parameters, impeller efficiency should be predicted within 
1% or less and head rise should be predicted within 5% or less. 

These criteria are summarized in Table 2. 


Table 2. Impeller Design Criteria for Three Validation Phases 


Criteria 

Design Phase 


Conceptual 


Detail 

Global 




Efficiency 

Qualitative 

1-2% 

<1% 

Head Rise 

Qualitative 

10% 

<5% 

Specific 




Velocity 

Qualitative 

±5% 

±1% 

Flow Angle 

Qualitative 

±1.0% 

±0.5% 

Flow Split 

Qualitative 

±5% 

±2% 


3.2 Code Validation Procedures Results 

Selected results of the four code evaluation demonstration cases are presented. 
Phase 2, 3, and 4 calculations used the k-e turbulence model. It is generally accepted 
that this model is sufficient to simulate turbulent flows where strong separation regions 
or shocks are not present. 

3.2.1 Phase 1 - Straight Duct 

Emphasis for Phase 1 was on verifying program logic, numerical error assessment, 
and the code convergence rate. It also reviews the code's capability in computing 
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flows using a multiple domain approach and examines a possible source of errors 
associated with the multizone grid approach. 

Ideally, a parabolic profile should be predicted and the centerline (maximum) velocity 
should be twice the average. Figure 5 shows the computed streamwise velocity at the 
centerline of the duct outlet using both coarse and fine grids. The coarse grid solution 
(12x6x6) underpredicts the centerline velocity and predicts the wrong the velocity 
profile shape due to insufficient grid resolution. The fine grid solution (26 x 22 x 22) 
correctly predicted the fully developed parabolic profile with the centerline velocity at 
two times the average velocity. The comparison in Figure 5, clearly indicates that 
even for the simple straight passage flow, sufficient grid resolution is critical in correctly 
predicting the flow characteristics. 

Two or more computational zones are often employed to model complex flowpaths. 
The grid must be smooth, not only within each zone, but across the zonal interfaces. 
The duct was regridded using two zones to study this effect. Figure 6 shows the 
computed centerline velocity at the duct outlet using both the single zone and two 
zone grids. In practice, the flow solver computes each zone separately and the 
information between each zone is communicated by proper interface boundary 
conditions. Although Figure 6 shows both approaches resulted in nearly identical 
velocity profiles, further examination of the flow characteristics in the full domain 
indicates the importance of a smooth grid distribution at the zone interface. Figure 7 
shows the velocity distribution at the duct midsection using smooth and nonsmooth 
grid interfaces. The solution with a nonsmooth interface grid shows a local 
discontinuity in the velocity contours. 

To further examine code logic, convergence histories were checked for single and 
multizone calculations. Figure 8 shows these convergence histories. The normalized 
residuals decrease by three orders of magnitude within twenty iterations for both 
calculations. Consistency between the two approaches shows the multizone 
approach to be logically sound. 
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X X 


NON-SMOOTH INTERFACE GRID SMOOTH INTERFACE GRID 

Figure 7. Velocity Contours for Nonsmooth and Smoother Zonal Interfaces 



Figure 8. Convergence History for 1- and 2-Zone Duct Flow 
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3.2.2 Phase 2 - Square Duct with 90° Elbow 

REACT was further validated by studying more complex flows. In this case the 
geometry begins to approximate that of the impeller. Complexities due to boundary 
layers and curvature-induced secondary flows are present. Addition of these 
important features increases the difficulty of accurately predicting impeller flows. 
Figure 9 shows the flow configuration. Figure 10 shows both the streamwise and 
radial velocity profiles in the spanwise direction at the 77.5° location for different radial 
cuts. The figure includes results of three numerical solutions with different grid 
distributions as compared with benchmark experimental data by Taylor, etal 13 . 
Several observations can be made: 

1 ) The calculations predict the right velocity profile (qualitatively) even with the 
most coarse grid (88 x 22 x 12) 


PLOW 




Figure 9. Flow Configuration for Square Duct with 90° Elbow 
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EXPERIMENTAL DATA 
80 x 22 x 12 GRID 
88x42x 22 GRID 
88x02x42 GRID 
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2) Increasing the number of grid points consistently improves agreement between 
the predictions and the test data 

3) While the coarse grid solution showed large errors in some locations (on the 
order of 30% at radial location R = 0.9), qualitative trends remain consistent 
with the data. 

The above observations further reinforce the code's capability and consistency. It also 
indicates the necessity to increase the grid density to capture the secondary flow 
associated with curvature of the passage. Further assessment of the effect of the 
secondary flow can be made by studying the associated numerical error distribution. 
Figure 11 shows the numerical error and secondary flow distribution at the 60° 
location. The location of the strongest secondary flow is consistent with the place 
where the largest numerical error occurs. It indicates that the secondary flow region 
requires finer resolution. An additional observation can be made. Although a given 
grid density (for example, 88 x 22 x 12) may be sufficient to resolve certain flows 
such as a straight duct, it may not be sufficient for computing other flows accurately 
(such as the 90° elbow). Systematic evaluation of numerical results through grid 
sensitivity studies and numerical error assessment should be used to select the proper 
grid distribution to accurately compute the given flow without unnecessarily increasing 
computing cost. 

3.2.3 Phase 3 - 3-D Turbine Cascade 

Solution of a turbine cascade flow introduces new geometric and flow complexities. 
Cascade flows are characterized by features such as high pressure gradients, end 
wall boundary layers, strong curvature, and secondary flow; many of the same flow 
features found in impellers. 

Resolution of the flow near leading and trailing edges is important for accurate 
aerodynamic loading and heat transfer predictions. Two topologies, each with coarse 
and fine grids were studied as a part of the Phase 3 effort. 'H' grids are most often 
used for these calculations as they are the simplest to generate. 'H' grid computations 
can predict reasonably good overall static pressure distribution, but accuracy usually 
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deteriorates in leading and trailing edge regions. A more elaborate 'O-H' grid 
topology was explored to help resolve this problem. A multiple domain grid was 
constructed using an ’O' grid to enclose the blade surface (helping to resolve the 
leading and trailing regions) and an 'H' grid outside. Construction of the multizone 
grid and making associated changes in the flow solver require additional effort, but do 
pay off in terms of increased accuracy. Figure 12 shows the single and multiple zone 
grids used for the SSME turbine blade. 

Figure 13 shows the static pressure distributions on H and O-H grids. In both cases 
the grid number used was 20 x 12 x 6. The multiple zone (O-H grid) calculation 
shows better agreement with test data. To evaluate grid sensitivity and performance 
error assessment, finer grid systems were constructed by doubling the grid number 
used in both streamwise and circumferential directions. While both the single zone 
and multizone solutions show improved agreement with test data (Fig. 14), the 
multizone O-H grid system appears to provide consistently better predictions. 

3.2.4 Phase 4 - SSME HPFTP 

Impellers are highly three-dimensional. Flows are dominated by strong curvature 
effects, high rotational speeds, strong pressure gradients, end wall boundary layers, 
and secondary flows. Figure 4 shows the selected SSME high pressure fuel pump 
impeller from the space shuttle main engine. The geometry for this impeller is very 
complex. There are three partial blades between every two main (full) blades. In 
order to calculate the flow inside this impeller, a multiple zone approach was used. A 
six zone flow solver was programmed into the REACT3D code and a 3-D six zone grid 
was constructed. This calculation was restricted to the impeller itself. The downstream 
crossover passage and the diffuser were not included in the calculation and 
interaction effects between these components were not taken directly into account. 

Figures 15a and 15b show 2-D planes of the impeller CFD model. Every attempt was 
made to include the significant features of the impeller and housing geometries. 
Figure 15a shows the expansion at the discharge of the impeller into the vaneless 
space. Figure 15b shows the blade-to-blade view of the impeller passage, note that 
the thickness at the discharge of the impeller vanes was maintained in the modeling. 
The grid generation was completed using an algebraic grid generation program 
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a. "H" Grid Topology b. "O-H" Grid Topology 

Figure 14. Static Pressure Distribution on Fine Grids 

developed at Rocketdyne for the design of impellers and inducers. The generation of 
the 30,000 grid point and 90,000 grid point meshes tool less than 3 minutes on an HP 
400 workstation. 

Two grid sizes were run to allow evaluation of the grid size requiremant for design. 
The larger the grid the longer it takes to obtain a converged solution. This also 
allowed determination of using a small grid size for preliminary design and then a 
larger for more detail design applications. As a basis for time convergence time 
requirements, the 30,000 grid took four hours to run on a HP DN10000 workstation 
and the 90,000 grid took 16 hours. 

The boundary conditions were set to embody the physical attributes of the impeller 
environment. Working from the inlet of the impeller, the boundary conditions are: 
stationary wall at the shroud with a small slip boundary just upstream of the impeller 
leading edge to mimic the gap between the stationary housing and the impeller and at 
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Figure 15. Impeller Grid Topology (30K Grid) 



the hub a rotating boundary; the walls of the impeller hub and shroud as well as the 
blade surfaces (not shown) were considered as rotating walls; at the discharge the 
lower surface of the expansion contained rotating wall boundaries to model the shroud 
and hub thicknesses and slip boundaries to model the gaps between the impeller 
shroud and hub and the stationary housing; the rest of the geometry is stationary and 
is modeled as such. 

One of the significant boundary conditions which was not modeled was the leakage 
flow down the hub and shroud surfaces at the discharge of the impeller. This was not 
done to simplify the CFD calculation. The effect of not modeling this flow is evident 
when the CFD results are compared with the test data. 

Validation of CFD for the design of new impellers requires that the code be capable of 
providing a reasonable prediction of both the flow at the immediate exit of the impeller 
and at a plane commensurate with the location of the downstream component. The 
important characteristics at the immediate discharge of the impeller are the relative 
velocity, flow angle and flow split between the impeller blades. At the downstream 
component, the absolute velocity and flow angle prediction are of concern. The 
evaluation of the flow comparison was made for both circumferentially averaged 
quantities and the detail flow field. The former allows evaluation of the averaged flow 
characteristics and the latter evaluation determines the usefulness of the CFD solution 
for the prediction of dynamic loads and forcing functions on the downstream 
component. 

Mass flow splits m each blade passage were calculated from the data and the CFD 
predictions. The significance of this evaluation is to provide for the design engineer an 
analytic capability for the placement of splitters in the flow field. Table 3 shows the 
results. The prediction of the flow split with both the 30,000 and 90,000 grid point 
models is within 1 .5%. This is within the accuracy required for even detail design 
purposes. 

Figure 16a shows the comparison of the circumferentially mass averaged relative 
velocity between the test data and the 30,000 and 90,000 grid point solutions at the 
plane immediately downstream of the impeller. The 30,000 grid point solution under 
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Table 3. Impeller Flow Split Comparison 


Flow Branch 

Test 

Data 

30,000 

Grid 

90,000 

Grid 

FULL Suction - Short Partial Pressure 

20.84 

21.31 

19.68 

Short Partial Suction - Long Partial 
Pressure 

26.48 

24.98 

25.99 

Long Partial Suction - Short Partial 
Pressure 

24.54 

25.97 

25.97 

Short Partial Suction - FULL Pressure 

28.14 

27.67 

28.37 


predicts the test data by approximately 30% within the b2 width region. The 
90,000 grid point solution underpredicts the test data by a maximum of 15%. It can be 
seen that the 90,000 grid point solution is better able to predict the trend across the 
impeller b 2 width than the 30,000 grid point solution. 

Figure 1 6b shows the comparison of the circumferentially mass averaged relative flow 
angle between the CFD solutions and the test data at the plane immediately 
downstream of the impeller. Although the flow angle magnitude prediction is very 
good within the most central region of the impeller b 2 width, the magnitude correlation 

breaks down in the outer regions. The trend is well predicted throughout. 

Figure 17a show the comparison of circumferentially mass averaged absolute 
velocities at the downstream plane. The prediction of both CFD solutions is within 
10% for the majority of the flow domain within the impeller b2 width. The 90,000 grid 

solution is much better able to pick up the trends in the flow field than the 30,000 grid 
solution. 

Figure 1 7b show the comparison of circumferentially mass averaged absolute flow 
angle at the downstream plane. The prediction is very good in both magnitude and 
flow angle for the entire flow region. The discrepancy in the region of the impeller 
shroud is attributed to secondary flows caused by the leakage down the impeller- 
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Figure 17. Comparison of CFD Solution to Test Data - Plane 3 
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housing shroud cavity which are not modeled. Both solutions are adequate for use in 
design of the leading edge of a downstream component. 

Figures 18-21 show further comparisons of test data with CFD predictions. Test data 
are shown for two radial locations downstream of the impeller discharge. Plane 1 is 
immediately downstream at a radius of 5.570 inches and Plane 3 is further away at a 
radius of 5.833 inches. Velocity and flow angle data are shown for various locations 
across these planes. The X values shown are normalized by the shroud to hub 
distance The shroud is located at X=0.0 and the hub is at X=1 .0. CFD predictions 
from the 30,000 and 90,000 grid point cases are compared in each case. General 
observations drawn from these comparisons include: 

1. Generally good agreement is achieved overall. 

2. Agreement within the passage region is consistently better than that outside 
where wall effects are significant. 

3. CFD predictions of velocity are reasonably good, particularly away from the 
walls. Increasing grid density improves the level of agreement. Clearly, the 
wake regions are missed to a large degree near the wall using the 30,000 point 
grid. Employing the 90,000 point grid improves agreement in the magnitude, 
but still misses the wake location. 

4. CFD predictions of flow angle are quite good. Increasing grid density improves 
the level of agreement, particularly near the walls. 
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Figure 18. Test Data versus 30K Grid Solution - Plane 1 
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Figure 21 . Test Data versus 90K Grid Solution - Plane 3 




4.0 SUMMARY 


A four phase procedure has been developed to standardize an approach for CFD 
code validation. The procedure is oriented toward the engineering design cycle. 
Three validation levels are defined to meet the needs of conceptual, preliminary, and 
detail design phases in the engineering environment. Detailed criteria established by 
the end user of the code results are used to judge the adequacy of code predictions for 
each validation level. 

The four phase procedure outlined utilizes a series of test cases, increasing in 
complexity, and always with a focus on the final application of interest. Phase 1 test 
cases are used to assess code numerics, verify its logic, and study fundamental 
operability. Phase 2 tests compare code results with benchmark quality experimental 
data to further test the physical models and understand necessary grid requirements. 
Phase 3 tests code operation on flowpaths similar to the final application of interest. 
These flowpaths are simplified to a degree that high quality test data may be available, 
but contain most of the geometric and flow features anticipated in the final application. 
Finally, Phase 4 tests the code operability on actual hardware of interest. 

The procedure has been demonstrated using the REACT code. The final application 
area selected was design of an impeller. Phase 1, 2, and 3 test cases were identified 
based on successive decomposition of the impeller flow characteristics and available 
test data. Ultimately, REACT was used to model the SSME HPFTP impeller. Two 3-D 
computations were performed using 30,000 and 90,000 grid points to represent the 
complete flow passage from one full blade to the next. Results were compared with 
extensive test data taken for the same impeller. Agreement is generally good to 
excellent and the REACT code has now been validated for conceptual, preliminary, 
and detail design of impellers. 
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